The invention relates generally to testing of computer hardware and/or software and, more specifically, to a method and system for performing regression tests in a state-dependent data processing or information handling system environment. In addition, the invention concerns an according system-independent regression test apparatus.
Known information handling or data processing systems are implemented by hardware, software or both. They often use predefined functions for processing input data sets of large data repositories, e.g. functions like ‘create account’ or ‘list account’ as those typically used in a banking environment. These systems are enhanced and changed very often during their development and life cycle wherein such activities may introduce errors in the existing functions insofar as originally used signatures (parameter definitions) become invalid. This fact generates an extensive need for regression tests.
Thereupon, regression tests are also used during software development, particularly when a software programmer or developer designs a new program or makes changes to an existing program. After any modification of a program it is not sufficient to test only the changed or enhanced part of that particular piece of program. The programmer typically subjects the program to a plurality of regression tests to ensure that the program performs its intended functions.
It is emphasized that testing changes to an existing hardware or software system is a more time consuming endeavor than testing a completely new system, in particular program code, because changes often inadvertently have some impact on other elements of the entire underlying system. Studies show that, compared to new code, changes of a program code are ten times more likely to contain bugs.
In such a software development scenario, U.S. Pat. No. 5,694,540 discloses a method and a system for regression testing of a recompiled main software program which allow to perform only a limited regression test that has the same level of testing as if the main program were tested using an expanded regression test that tests virtually all of the software modules forming the program. More particularly, if an error is found in the main program, it is documented in a modification request which a programmer thereafter uses to track down and identify the software module(s) that needs to be modified to deal with the error. At that point, the programmer modifies the identified software module and also designs a current test program to test the recompiled program to determine if the error has been corrected and that the modification does not affect some other aspect of the program. The modified module and test program are then associated with an identifier identifying the associated modification request. The facility then locates all modification request identifiers respectively associated with prior modification made to the identified software module and executes the test programs associated with those modifiers as well as the current test program against the recompiled version of the main program.
More particularly, the above approach provides a test management program connected to a test case database that creates a test template. Uncovered errors are documented and identified as so-called ‘Modification Requests’ (MRs). The programmer modifies the source file(s) of the program in order to correct the identified errors. The programmer also designs a test that may be applied to a master program when it is recompiled to substitute the modified source file(s). The test is designed to test the master program to determine if the errors documented in the MR is actually be corrected. In particular, it is therein proposed that only those test cases associated with the source file(s) that have been changed and thus associated with the corresponding MR(s) are invoked to regression test the recompiled master program. The mentioned process of updating the source code is done parallel with the updating of the test code.
In addition, a status information table is updated using the test results i.e. which test cases passed and which test cases failed.
In addition, a test unit must be re-run if any of the program entities it covers has changed. However, it is difficult to identify the dependency between a test unit and the program entities that it covers. In order to overcome these obstacles, selective retesting techniques have been previously described in the literature, e.g. Yau, Stephen S. and Kishimoto, Zenichi, “A Method for Revalidating Modified Programs in the Maintenance Phase”, in Proceedings of the 11th Annual International Computer Software and Applications Conference (COMPSAC), pages 272–277, IEEE Computer Society, Oct. 7–9, 1987 which describes a selective retesting technique for partition testing.
Another approach in the field of software engineering is known from U.S. Pat. No. 5,673,387 which discloses a system and method for selective regression testing of a software system that determines which test units or subset of a test suite must be re-run after a modification to or a new version of a software system. The entities which are covered by each test unit are identified. When the software system is modified the entities which were changed by the modification are identified. The test units which need to be re-run are determined by analyzing the change information and the coverage information to select those test units that cover changed entities. When the software is changed, the set of changed entities is identified. This set is then compared with each set of covered entities for the test units. If one of the covered entities of a test unit has been identified as changed, then the test unit is re-run. Hereby a user generates a list of changed entities to determine which test units must be re-run in the case of a hypothetical system modification.
Thereupon, complex and mission-critical business applications typically contain a large number of on-line transactions, each of which consisting of considerable functionality. The software application quality therefore largely depends on extensive testing of the functionality of each of the on-line transactions to ensure completeness and correctness. Traditional automated application testing strategies have only been partially successful in minimizing the manual nature of transaction testing. Typically a keystroke-capture technique is employed wherein the computer memorizes the keystroke and mouse movements performed by a test operator enabling the computer to replay the test at a later time for retesting the application. But the test data generated by the keystroke-capture technique is normally non-maintainable and incomprehensible, and therefore, any changes to the test data require re-creation of the test to enable the entire test to be re-recorded for subsequent playback.
As already mentioned, most regression tests in such a scenario rely on a large set of test data. To repeat the tests, complex systems require an update of the test data in order to adapt them to a new functional context of the system.
In particular, another shortcoming of traditional automated testing techniques is preparation of the test data. Once the test data is developed, maintaining such test data is virtually impossible due to the manually developed origin. It is known, that application programmers heretofore create non-meaningful test data that often does not actually properly exercise the transaction or application undergoing the testing process.
A corresponding method of regression testing of transaction based software applications during the development and other life cycle phases of the software is disclosed in U.S. Pat. No. 6,061,643. Regression tests comprised of test cases containing test data describe the target test at a functional or behavioral level and executes a regression test at a physical level. The functional level accommodates changes to the transaction such as the moving of text fields to other locations within a frame or the changing of the presentation of particular fields from one form to another. A test operator performs a manual test and simultaneously records the test. The test data is in a robust functional description of the transaction such that physical modifications to the transaction during software development preserve the viability of the test data for execution in the modified transaction. A particular component facilitates the creation of test cases for transactions by monitoring the performance of the test in the transaction itself. A test report is compared with a control test report to verify the lack of regression of the transaction. Accordingly, changes to the transaction therefore allegedly shall not result in unusable test data.
The above approaches have in common the drawback that there is a large number of pitfalls which are expensive to uncover by experience and that they involve large test sets of unknown content, lots of redundant tests, and bugs that slip through even though we supposedly used regression testing to eliminate the side effects of changes.
In addition, the known approaches can not automatically perform regression tests in a state-dependent data processing system, e.g. a transaction system comprising unpredictable states at the time of the regression test and being non-resettable to an initial state at the beginning of the test, thus requiring considerable user interaction or a special test program supporting the underlying logic for handling such unpredictable state changes. While executing the test, the state of each subtask may be unpredictable for some required parameter values as well.